2 links tagged with all of: superintelligence + recursive self improvement
Click any tag below to further narrow down your results
Links
Rux Andrea Bio, writing after three weeks in San Francisco, reports that many AI lab insiders and safety researchers see "handover" to a superintelligent AI as an acceptable or even good outcome. She argues the vision is vague, undemocratic, and closer to dystopia than utopia.
- Some researchers, including a lab builder she quotes, see humanity ceding control to a superintelligence after recursive self-improvement as likely and tolerable, with p(doom) estimates of 10 percent or higher circulating.
- Handoverism is loosely defined: the author estimates 30-40 percent of the outside AI safety community holds some version of it, with prominent figures like Scott Alexander favoring a benevolent "God" AI.
- The author argues handover offers no clear account of democracy, consent, labor, or purpose for humans, so even "happy" scenarios may lose most of what people value.
+ handoverism
+ scott alexander
recursive self improvement
superintelligence
+ ai safety
+ pdoom
+ singularity
The author argues that running millions of AI agents in parallel won't easily produce a self-sustaining intelligence explosion, because the data show parallel agents help with speed far more than with solving harder problems. Gains from more agents tend to flatten fast, and the piece treats OpenAI's Navier-Stokes run as a reminder that agent count isn't the same as useful output.
- More parallel agents mainly speed up work; they add much less to intelligence on hard problems, and gains often diminish quickly or saturate.
- The author sorts diminishing returns into a ladder (power-law, logarithmic, sub-logarithmic), and says the clearest multi-agent evidence looks logarithmic or flatter, too weak to sustain runaway self-improvement.
- Three limits drive this: coordination ability, homogeneous thinking among copies, and how much of a problem can actually be parallelized.